Supress alert on tags

#1

Hello,

I want to distinguishes two kind of alerts:

  • alerts that must be handled everytime: office hours, non office hours, day, night, weekend, days off
  • alerts that must be handled only in office hours

I want that office-hours-only alerts does not triggers notifications during night oncall, weekend
even better if I can auto suppress these alerts.
said otherwise:
If alert has tag office-only and current schedule is office-hours => then supress alert

I have Sensu monitoring plugged in to PagerDuty via API integrations.

I want to keep scheduling intelligence only in PagerDuty. I don’t want to configure Sensu with scheduling also
I can configure tags like “office-hours-only” on Sensu checks

Any ideas ?
Service event rules seems a good place but it misses the possibility to have a condition with current schedule

Thanks a lot

#2

Hi Axel,

I’m on the product team here at PagerDuty. Have you already checked out the support hours functionality in PagerDuty? This lets you set up different notification behaviors based on time of day / day of week.

I think we do have an opportunity to better integrate this functionality with service event rules, but I wanted to know if you’ve already tried out support hour settings and if so, what the biggest gaps you’re seeing are.

#3

I checked this functionality.

First, We already have Schedules for support hours. So, I don’t understand very well the point of re-configuring the support hours here

Second, I’m looking for: On support hours=> notify, after support hours => do NOT notify anyone. I don’t see this possibility with the tool you mentionned.

Third, My main problem is to categorize the alerts coming from the same system, independentely from their severity.

I think I will split into 3 service for the same monitoring system.
and bind alerts to a particular service in my monitoring system.

I hope you understand what I am looking to build. it’s quite difficult to explain workflow by text !

#4

Hi Axel,

Starting with Sensu Core (Agent-based) method of integrating with PagerDuty and modifying it a bit, it should possible to integrate Sensu with multiple PagerDuty services by setting up the same integration more than once, with different integration keys, and thus allow you to achieve the desired effect. Under this configuration, scheduling and escalation would still be handled in PagerDuty.

It would work through allowing you to distinguish alerts that should go to one service with support hours (and alert during hours / produce low urgency or no notifications after hours, depending on the agents’ notification settings) from alerts that go to another service that notifies on-call agents at any time. This would be achieved by assigning different Sensu handlers to the appropriate checks.

The way one would do this is to create several differently-named check handlers that call pd-sensu with a different integration key, so instead of pagerduty you might have pagerduty_urgent and pagerduty_supporthours.

You’d start in step 2 of the guide by making two aptly-named copies of the configuration template, i.e. pagerduty_urgent_handler.json and pagerduty_supporthours_handler.json, and in each one of them change the name of the handler.

For instance:

wget -O /etc/sensu/conf.d/pagerduty_supporthours_handler.json https://raw.githubusercontent.com/PagerDuty/pdagent-integrations/master/conf.d/sensu_pagerduty_handler.json

vi /etc/sensu/conf.d/pagrduty_supporthours_handler.json

{
  "handlers": {
    "pagerduty_supporthours": {
      "type": "pipe",
      "command": "/usr/share/pdagent-integrations/bin/pd-sensu -k YOUR-INTEGRATION-KEY-HERE"
    }
  }
}

The final step will be to fill in the handler configurations with the different PagerDuty integration keys.

Please give that a try, and let us know if it works!